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Abstract: Most of the VLE design methods focus on producing content for a VLE. However, usability of the VLE 
is also of great importance. Several potential usability problems have been reported in recent e-learning 
conferences. These problems could have been avoided by applying usability engineering methods before the VLE 
was taken into use. This paper describes usability engineering methods used to ensure the usability of Virtu, a 
virtual learning environment (VLE) concerning safety at work. The results of using each method are summarized 
and as a conclusion, some general VLE design guidelines are listed to help others in VLE design. 
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1. Introduction 

Most of the VLE design methods focus on 
producing content for a VLE. However, 
usability of the VLE is also of great importance. 

If the user interface of a VLE is not usable, the 
user’s focus on the actual content is 
diminished, because using the VLE requires a 
considerable amount of concentration. Good 
usability, on the other hand, allows the user to 
focus on the content thus improving learning 
results. 

Several potential usability problems have been 
reported in recent e-learning conferences. 

Remarkable obstacle to taking VLEs into use 
rises from the users' background knowledge of 
computer technology (Jones et al, Kent 2003). 

The idea of distance-learning brings social 
problems like impersonal feeling and non- 
alignment for the course (Lindh and Soames, 


Kent 2003). Ever increasing VLE sizes make 
the VLEs disorganized and therefore a 
navigational aid is needed (Armitage et al 
2003). The high enthusiasm on latent potential 
of VLEs makes the designers implement as 
many features as possible. However, the VLEs 
may thus contain unnecessary features that 
students never or rarely use (Beasley and 
Smyth 2003). 

Virtu is a virtual learning environment (VLE) 
concerning safety at work (Figure 1). Virtu was 
developed in a co-operational project between 
the Institute of Occupational Safety 
Engineering and Institute of Software System s 
in Tampere University of Technology (TUT) . In 
addition to more traditional VLE content, such 
as theory and exercises, Virtu contains a 
virtual enterprise, which simulates a visit to a 
real company. 


Introduction 


Main menu 


Textbook section 


Virtual enterprise 



, section 


Theory 


Exercises 

i_ 


r 

Funtion points 

l,__ 


Enterprise presentation 


Figure 1: Structure of Virtu 


The early design phase of Virtu was done 
according to ISO 13407 Human centred design 
processes for interactive systems (ISO 1999). 


The standard includes four user centred 
design activities to be started at the earliest 
stage of a project. These design activities 
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resulted in defining two main usage scenarios 
related to two main end user groups for Virtu. 

The first scenario is about teaching safety at 
work for students of Occupational Safety 
Engineering at TUT. The second scenario is 
testing and updating industrial workers’ 
knowledge of safety issues related to their 
work. Virtu user interface was designed 
applying usability practices including 
stakeholder and user interviews, prototyping, 
heuristic evaluation, iterative design and user 
testing so that both end user groups would find 
the environment interesting and easy to use 
(Preece et al. 2002). Virtu supports 
constructivist learning model where individuals 
are assumed to learn better when they 
discover things themselves (Ahmad et al. 
1998). 

This paper describes usability problems of 
current VLEs. Usability methods used during 
Virtu development process are also outlined 
and, as a conclusion, some general VLE 
design guidelines are presented. Potential 
usability threats of current VLEs reported in 
publications are discussed in Section 2. 
Section 3 presents usability engineering 
methods used to ensure usability of Virtu and 
the summarized results of using each method. 
Virtu learning environment produced by the 
software and usability engineering activities is 
outlined in Section 4. Section 5 provides 
general VLE design guidelines, which rose 
from Virtu development process. 

2. Usability problems of current 
VLEs 

In their study Jones et al (2003) identified eight 
primary causes of student withdrawal. Three of 
these causes were to some extent technology 
related. Technical problems as such were 
reported as a cause of withdrawal by 20% of 
the respondents. Also, own IT skills were 
identified as a reason for withdrawal by 15% of 
the respondents. Moreover, the use of 
technology had an impact on student 
confusion and lack of understanding, which 
contributed to 15% of the respondents’ 
withdrawals. Also Kent (2003) reports on 
students having had difficulties in using a VLE. 
The lack of technical skills necessary for 
understanding or participating in activities 
within the VLE was reported to be one of the 
features the participants did not like about the 
e-Learning module (Kent 2003). 

Another aspect that propagates the student 
withdrawal from a course is the impersonal and 


faceless nature of VLEs (Kent 2003). Lindh 
and Soames (2003) state that there is still a 
need for human-to-human interaction. For 
example, asking questions is much more time 
consuming in purely electronic environments 
than in face-to-face contact in a classroom. 
When more and more students are part time 
students having work career and family life at 
the same time, the challenge is to activate 
students and to make them to commit 
themselves to a course. 

Students’ interests towards a course can 
collapse because of muddled and ambiguous 
electronic material. Identifying content can be 
a problem (Lindh and Soames 2003). For 
example, a document containing multiple 
pages may come out of printer without page 
numbers or pages have irrelevant title or no 
title at all so they cannot be connected to the 
topic later on. The level of navigational 
freedom affects to the user’s feeling of control 
(Armitage et al. 2003). Navigational aids 
providing higher navigational freedom (maps, 
indexes) give higher feeling of control 
compared to navigational aids providing lower 
navigational freedom (back and forward 
buttons, hypertext). Even navigational aids do 
not guarantee the optimal use of learning 
material, but instructions on how to study non¬ 
linear hypertext material are also needed 
(Beasley and Smyth 2003). 

According to Beasley and Smyth (2003), 
students often interact with VLEs in a manner 
that will not allow them to fully exploit the 
VLEs’ potential. In their paper concerning 
students’ selective use of a virtual learning 
environment, Beasley and Smyth (2003), in 
addition to their other results, reported on VLE 
features which were either rarely used or were 
not used at all. These features included 
individual progress report and asynchronous 
discussion forum. 

To relieve the students’ stress caused by 
online activities, Lawless and Allan (2003) 
recommend that the technology used should 
be user-friendly and errorless and should 
provide a pleasant experience for the user. 
Another aim of VLEs is to provide cost 
effective education (Lindh and Soames 2003). 
By designing and testing the VLE according to 
usability engineering methods before taking it 
into use, all these objectives can be achieved. 

3. Usability engineering 

In the beginning of the iterative design 
process, a paper mock-up of Virtu user 
interface was developed based on stakeholder 


http://www.eiel.org 


©Academic Conferences Limited 



105 


Heli lhamaki & Inka Vilpola 


interviews. Usability tests with real end users 
were conducted using the paper mock-up and 
the test users were also interviewed. Based on 
the test results, a software prototype of the 
user interface was developed and usability 
tests with end users performed. After the 
second usability testing round, a new software 
prototype was designed and heuristic 
evaluation was conducted by an usability 
professional. 

As a result there was list of changes to be 
made in every area of design: structure, 
content and user interface. Because the 
shortcomings were detected before the 
implementation was completed, they could be 
corrected before the final release version. 
Thus, iterative design process diminished the 
need for changes in the future. 

3.1 Human centred design 

Requirements were formed by specifying the 
context of use (UsabilityNet) and thereby the 
purpose of use. Also the user and 
organizational requirements were collected 
trough stakeholder interviews (UsabilityNet). 
Technical limitations and updates must be 
considered in early design phase, too. 

The stakeholders of Virtu were specified first 
by listing all the factors affecting the use of 
Virtu and thereby the people related. Virtu’s 
stakeholders included end users, teachers in 
the Institute of Occupational Safety 
Engineering, maintenance, marketers and 
those who are responsible for the safety at 
work issues in industry. 

3.1.1 User and organisational 
requirements 

Virtu has two main end user groups. The first 
end user group is students of Occupational 
Safety Engineering at TUT and the second 
group is industrial workers. The computer skills 
and the requirements for the content of Virtu 
differ greatly among these groups. 

The approach to safety at work issues should 
cover all the legislation concerning safety and 
health at work. It should be challenging 
enough for students who already have a wide 
knowledge of the subject. However, the 
content cannot be too theoretical, because the 
learning environment is to be used by 
industrial workers, too. They do not necessarily 
have a wide knowledge basis on the issue but 
are more interested in practical ways of 
improving the safety of their own work. 


The human factor goals of end user groups 
can be established according to 
Schneiderman’s five measurable human 
factors: 

■ time to learn, 

■ speed of performance, 

■ rate of errors by users, 

■ retention over time, 

■ subjective satisfaction. (Schneiderman 
1998) 

The priorities of the human factors vary 
between students and industrial workers. 

For students time to learn and retention over 
time are the most crucial factors. Students use 
Virtu instead of traditional course material, and 
it is supposed to be as intuitive to use as a 
book and a pen. The usage time is limited to 
only few hours a week as if there were 
traditional lectures. The knowledge of how to 
use Virtu must be maintained between 
lectures. 

Retention over time is even more important to 
industrial workers, who may have an intensive 
course on safety at work, and are returning to 
Virtu after long period of time. Subjective 
satisfaction counts when the use of Virtu as a 
reference book is discussed. 

3.1.2 Context of use 

The context of use turned out to be complex. 
At least three environments and three different 
purposes of use were defined. Users should 
be able to use Virtu independently at home or 
at work, and, in addition, by groups in safety at 
work training. There should also be a 
possibility to use Virtu as a quick reference 
book. All these different usages must be taken 
into account in the user interface design. 

In most of the cases Virtu is used for studying 
occupational safety engineering at university, 
training safety at work issues related to 
everyday work in industry or revising issues 
related to safety at work. Students are enrolled 
in a course dealing with safety at work. 
Industrial workers are participating in a course 
arranged by employee or otherwise are 
recommended or required to get familiar with 
safety at work material provided by Virtu. 

Students will mainly use Virtu in computer 
classrooms with internet connections. 
Computers usually include Pentium 500 MHz 
processors with 128 MB memory. However, 
high processor and memory capabilities cannot 
be expected from computers used by industrial 
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workers at their workplaces, and in that 
environment internet is usually not allowed. 
Both user groups can use the Virtu also at 
home, but those environments cannot be taken 
into account in the design process. 

3.1.3 Technical limitations and updates 

Virtu will be a commercial product and 
therefore should not be used freely in the 
internet. Instead, some kind of a copy 
protection or license practice in its distribution 
must be implemented. CD-ROM was chosen 
as the distribution media. Distribution via 
internet was rejected due to the requirements 
of participating companies. In the future, Virtu 
might be used through a web browser, and 
therefore there should be a possibility to 
change the user interface to a web based 
version without changing the application logic. 

Legislation concerning safety and health at 
work changes from time to time and also other 
new information on the safety at work issues 
may emerge. To ease the introduction of 
updates, changing the content must not cause 
changes in the user interface. Also, changing 
the content should be possible without 
programming. Internationalizing Virtu for 
Western cultures should be easy, too. 

3.2 Usability tests 

As Virtu was designed according the ISO 
13407 Human centred design processes for 

interactive systems (UsabilityNet, ISO 1999), 
the proper usability engineering methods 
including iterative design, user interviews, 
prototyping, user testing and heuristic 
evaluation, were used to gather and verify 
Virtu's properties. 

3.2.1 Iterative design 

Virtu user interface was developed based on 
an iterative design process, which means that 
new versions of the user interface were 
produced based on the usability problems and 
opportunities disclosed by empirical testing 
(Nielsen 1993). During the iterative design 
process the interface was evaluated by end 
users and usability professionals. 

Iteration began with user interface drafts drawn 
in a paper. There was one paper for each type 
of displays; menus, theory, exercises, virtual 
enterprise, function points and description of 
the enterprise. After end users had tested the 
paper version, a software prototype was 
developed. In the first phase only menus and 
one case of each functionality was 


implemented. After user tests, the first full 
version was built for further testing. 

3.2.2 Stakeholder and user interviews 

Many aspects of usability can be studied by 
simply asking the users. This is especially 
useful for issues that are related to users’ 
subjective satisfaction and possible anxieties 
(Nielsen 1993). In the beginning, when nothing 
was implemented, the stakeholders of Virtu 
were interviewed. The analysis of results of the 
interview produced requirements for the first 
paper prototype of Virtu. During Virtu usability 
test with a paper mock-up, the users were 
asked to fill two interview forms. 

The first form was filled in the beginning of the 
test. It contained open and multiple-choice 
questions concerning users' computer usage 
and demographic information. The age of 
participating students varied from 20 to 29 and 
the age of participating industrial workers from 
27 to 53, being very typical to both user 
groups. The sex ratio in interview was 4/5 
females in students and 2/6 females in industry 
workers. 80-83% of both groups used 
computer about 3 hours per day. Most of the 
students but clearly a minority of industrial 
workers used computer also for leisure. 2/6 of 
the industrial workers used operations and 
work management systems in their work. 

The second form, filled at the end of the test, 
contained five open questions. This 
questionnaire was about the ease of use of 
Virtu and the users’ subjective opinion about it. 
The results indicated that all students but only 
half of the industrial workers considered Virtu 
to be easy to use. The main reasons for finding 
Virtu difficult to use were navigation problems. 
More than half of the students and less than 
half of the industrial workers said to be 
interested in using Virtu for studying purposes. 

3.2.3 Prototyping 

Prototypes are useful for conducting system- 
specific usability tests early in the development 
process. Their role is to demonstrate and test 
aspects of human-computer interaction, which 
might be difficult or more time-consuming to 
produce in another medium (Lindgaard 1994). 
Especially low-fidelity prototypes like paper 
mock-ups are useful, because they are simple, 
cheap and easy to produce and thus easy to 
modify for testing alternative design ideas. 
Software prototypes as high-fidelity prototypes 
are useful in testing out technical issues 
(Preece et al. 2002). Both paper mock-up and 
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a software prototype were tested during Virtu 
development. 

3.2.4 User testing 

User testing with real users is the most 
fundamental usability method, because it 
provides direct information about how people 
use computers and what their exact problems 
are with the concrete interface being tested 
(Nielsen 1993). During Virtu development, 
formative evaluation was carried out in order to 
improve the user interface as a part of iterative 
design process. Formative evaluation was 
performed as a typical thinking-aloud test. 

The major finding of the usability tests with the 
paper mock-up was the need to divide the 
content to different difficulty levels to match the 
needs of different user groups. In the 
beginning of the tests, there was only one 
common content, which most of the industrial 
workers found quite irrelevant because it was 
too theoretical compared to their work. Another 
important finding was the users’ difficulty in 
navigating in the virtual enterprise section. A 
ground plan of the virtual enterprise with 
marked function points, for example Gate or 
Check in, was used for navigation. The users 
did not recognize the function points as 
buttons but were wondering what they were 
supposed to do with the ground plan. 

There were also some minor findings. 
Navigation from one page to another in the 
textbook section was considered somewhat 
difficult by some industrial workers, because 
there were only arrow-buttons without a text to 
indicate where the button leads to. Most users 
considered the link from the virtual enterprise 
section to theory topics quite unnecessary. 

Usability testing with the software prototype 
only provided some minor findings. Some of 
the function points in the virtual enterprise 
section contain questions, for example "Do you 
know the nearest emergency eye rinsing at 
your workplace?”, for industrial workers to 
consider at their own working environment. 
The users found the visibility of such points on 
the ground plan to be quite weak and they also 
thought the questions were not separated 
clearly enough from the rest of content of the 
function point. Also the terms in the main menu 
were considered to be quite ambiguous, and it 
was not clear to the users what each section 
contains. 


3.2.5 He uristic e valua tion 

Heuristic evaluation and user testing should be 
alternated, because these two usability 
practices have been shown to find fairly distinct 
sets of usability problems (Nielsen 1993). 
Heuristic evaluation is done by studying a user 
interface and then forming estimation about 
what is good and bad about it. The evaluation 
is conducted according to a certain set of 
rules, such as those listed in typical guidelines 
documents (Nielsen 1993). In Virtu heuristic 
evaluation, the list of Nielsen's ten usability 
heuristics (Nielsen 1994) was used. 

Nielsen’s ten usability heuristics include: 

■ elements in user interface (4 heuristics), 

■ language (2), 

■ interaction (2), 

■ error prevention (1) and 

■ helping information (1). 

User should have appropriate feedback in time 
and the system status should be clearly seen. 
All the error descriptions and Ul terminology 
should use users language and terminology. 
Consistency can be maintained for example by 
using standards and Ul guidelines. Operation 
is made easy and efficient by providing 
accelerators, shortcuts, exits and easy 
escapes. The interface should contain all the 
essential elements but nothing more. (Faulkner 
2000 ) 

Heuristic evaluation was conducted for each 
display and for Virtu as a whole after user tests 
with the software prototype in order to search 
for usability problems undiscovered by user 
testing. Usually the heuristic evaluation is 
carried out before user tests, but in this 
iterative process it was reasonable to use the 
usability professional to verify the design 
decisions before release version of Virtu. 

Results indicated that some terminology like 
ground plan and enterprise description had to 
be changed into more user oriented 
vocabulary like visit and table of contents. 
Easy exits had to be added in the theory 
section, and the cursor had to be changed in 
appearance, when placed over buttons to 
show to the user the possibility of interaction. 
Some minor notes were reported about 
consistency among different parts of Virtu. 

4 . Virtu learning environment 

The structure of Virtu was split into two entities 
because of the requirements (Figure 1). Some 
educational material concerning safety at work 
already existed, and it needed to be presented 
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in electronic form in order to utilize the 
interaction possibilities of VLE. It was also 
necessary for both user groups to see some 
real life examples of safety at work. The 
practical part should also support the 
adaptation of theory. 

The two parts of Virtu are: 

■ textbook section, 

■ virtual enterprise section. 

The textbook section was divided to theory and 
exercises. The virtual enterprise section 
contains interactive function points and a 
description of the enterprise providing 
background information on the virtual 
enterprise. The material provided in function 
points was gathered from three real life 
companies. 

4.1 Textbook section 

The textbook section can be studied as a 
guided tour including theory and exercises on 
every topic. Each theory topic can also be 
studied independently from the others and the 
textbook section can thus be used as a quick 
reference book. The use of textbook section 
begins by selecting one of 8 topics covering 
different areas of safety at work. 

The content of the textbook was divided to two 
difficulty levels based on usability test results. 
Theoretical level contains information on 


different safety at work topics including 
legislation, occupational safety management, 
risk management, ergonomics, occupational 
hygiene, machine and system safety, mental 
welfare and occupational accidents. This level 
is to be used in industry by the people 
responsible for safety at work issues and by 
the students, because they need a deeper 
understanding of the issue. 

Practical level is to be used by industrial 
workers. Thus, the topics included differ 
slightly from the content of the theoretical level. 
The practical level contains information on risk 
assessment, ergonomics, occupational 
hygiene, occupational accidents, signs and 
traffic. Included information can be easily 
applied into practice in the user’s own work 
environment. Even the topics, which are same 
or similar to ones in the theoretical level, were 
re-designed to match the requirements of the 
group using this level. 

4.1.1 Theory 

Theory displays were designed using a book 
metaphor (Figure 2). Each view reminds 
opened book, which consists of two pages. 
The user’s location in the Virtu is indicated on 
the top and there is always possibility to exit 
the current theory section. User is informed of 
the total number of pages included in the 
section. 



Figure 2. Theory display 
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User navigates between theory displays using 
back and forward buttons at the bottom of the 
pages. Menus in the bottom of the display are 
consistent through the program. The user is 
expected to read the content as traditional 
textbook content. The main difference 
compared to traditional textbooks is the 
amount of graphics and the length of text 
chapters. Text chapters are short because 
reading is 25% slower from screen than it is 
from paper ( Nielsen 1997 ). 


4.1.2 Exercises 

The textbook contains seven different types of 
exercises including for example multiple-choice 
questions, drag and drop exercises, and 
crosswords (Figure 3). Feedback on the 
correctness of the user’s answers is given in 
graphical form. Also exercises on each topic 
can be used independently. Thus, Virtu can be 
used to test the user’s knowledge of safety at 
work issues for example in the beginning of a 
safety at work training session or to test 
learning at the end of such session. 



Figure 3. Multiple-choice exercise display 

In the beginning of the exercise page user is collected from real companies makes the 

guided how to carry out the exercise. There is virtual enterprise realistic. 

a possibility to check if the answers are 

correct. In the case of an error the right 

answers are not displayed, but the user has to 

revise them based on the theory part. 

4.2 Virtual enterprise 

The virtual enterprise is to be used by both 
user groups. Its purpose is to simulate a visit to 
a real company and to spread good safety at 
work practices to industry. The content of the 
virtual enterprise includes photographs, 
pictures, text and video and it has been 
integrated of three different industrial fields 
(metal, paperboard and plastic). Material 
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Figure 4. Ground plan used for navigation in the virtual enterprise 


User navigates in the virtual enterprise by 
clicking the marked function points on the 
ground plan (Figure 4). The function points are 
buttons through which additional information 
on the subject can be found. Some of the 
points contain safety at work questions for 
industrial workers to consider in their own 
working environment. 

4.3 Implications of usability 
engineering 

As an outcome of paper mock-up testing and 
interviews, some changes were made to both 
content and the user interface of Virtu. 
Because the user test results showed a need 
for different difficulty levels in the content, the 
people responsible for developing it were 
informed, and some new content with a more 
practical approach to the issue was thus 
developed. 

Navigation in the virtual enterprise was made 
easier by adding instructions to the ground 
plan display. Also the marking of function 
points was improved to make them easier to 
recognize. Some texts, which might disturb the 
user, were removed from the ground plan so 
that then only texts left were those of function 
point buttons. The link from the virtual 
enterprise to theory was removed, and 


description texts were added to the arrow 
buttons used in navigation. 

Usability testing with the software prototype 
resulted in some minor changes in the user 
interface. Visibility of function points containing 
questions for industrial workers was improved 
in the ground plan by changing the icon, and 
the distinction between the questions and 
other content was made clearer by adding a 
title for the questions and marking the title with 
the same icon as on the ground plan. 

The main implication of heuristic evaluation 
was that the virtual enterprise section was 
divided to two different sections already in the 
main menu. Also the terms in the main menu 
that were difficult to understand were changed, 
so that they would be clearer to the user. 
Changing the cursor when the cursor is over 
some button was also included. 

However, the possibility to jump to the table of 
contents while the user is in the middle of a 
theory section or a function point was not 
provided. The rationale for this is that in a case 
of a VLE it is sometimes appropriate to limit 
user freedom to improve learning (Alessi et al. 
1991). 
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5. Conclusion 

The experience of developing Virtu can be 
presented in three guidelines of VLE design: 

■ Specify the stakeholders of the VLE. 

■ Specify how the VLE is used. 

■ Apply usability engineering methods in 
designing and testing phases before 
taking the VLE into use. 

User groups can be defined as a wide range of 
stakeholders or as the end users of the VLE 
only. The stakeholders could be for example 
maintainers of the VLE or teachers responsible 
for teaching the content of the VLE to students. 
The advantage in wide range of users is a 
design covering all areas related to the use 
and lifecycle of VLE. 

The purpose for using the VLE can vary from 
being like a textbook or scientific material to be 
more like demonstration or even a game. The 
VLE may be used only once or the users may 
refresh their memory with VLE when needed. 
In some cases there can be many users 
simultaneously. They may all use one 
computer together or interact through 
networked computers. 

As technical difficulties may cause 
dissatisfaction resulting in student withdrawals, 
the technical skills of the potential users should 
be accounted for, and the user interface 
should be designed respectively. Unnecessary 
features increase implementation and 
maintenance costs. Therefore, the features 
included in a VLE should be carefully 
considered according to the usability 
engineering results in designing phase. Many 
of the usability problems in current VLEs could 
have been avoided by applying usability 
engineering methods before taking the VLE 
into use. 
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